/**
 * Copyright (c) 2016 NumberFour AG.
 * All rights reserved. This program and the accompanying materials
 * are made available under the terms of the Eclipse Public License v1.0
 * which accompanies this distribution, and is available at
 * http://www.eclipse.org/legal/epl-v10.html
 *
 * Contributors:
 *   NumberFour AG - Initial API and implementation
 */
/**
 * <p>
 * This package contains customization to massage the parser in order to consume valid JS files and detect invalid ones.
 * </p>
 *
 * <p>
 * These customization includes the automatic semicolon insertion as well as the disambiguation of regular expression
 * literals and division operations. Furthermore a special
 * {@link org.eclipse.n4js.parser.PropertyNameAwareElementFactory element factory} is used to keep track of the
 * concrete syntax of property names. Also included is a special
 * {@link org.eclipse.xtext.parser.antlr.SyntaxErrorMessageProvider SyntaxErrorMessageProvider} to handle value
 * conversion results properly.
 * </p>
 *
 * <h2>Disambiguation of regular expressions and binary operations</h2>
 *
 * <p>
 * The parsing process can basically be divided into three steps. First of all, the file contents has to be read from
 * disk. This includes the proper encoding of bytes to characters. The second step is the lexing or tokenizing of the
 * character stream. A token is a basically a typed region in the stream, that is a triplet of token-id, offset and
 * length. The last step is the parsing of these tokens. The result is a semantic model that is associated with a node
 * tree. All necessary information to validate the model can be deduced from these two interlinked representations.
 * </p>
 *
 * <p>
 * Since the default semantics and control flow of Antlr generated parsers do not really fit the requirements of a fully
 * working JavaScript parser, some customizations are necessary. Regular expression literals in JavaScript cannot be
 * syntactically disambiguated from div operations without contextual information. Nevertheless, the spec clearly
 * describes, where a regular expression may appear and where it is prohibited. Unfortunately, it is not possible to
 * implement these rules in the lexer alone, since it does not have enough contextual information. Therefore, the parser
 * has been enhanced to establish a communication channel with the lexer. It announces when it expects a regular
 * expression rather than a binary operation.
 * </p>
 *
 * <p>
 * This required a reworking of the Antlr internals. Instead of a completely pre-populated
 * {@link org.antlr.runtime.TokenStream}, the parser works on a lazy implementation that only reads as many characters
 * as possible without a disambiguation between regular expression literals and divide operators.
 * </p>
 *
 * <p>
 * Only after the parser has read this buffered tokens and potentially announced that it expects a regular expression,
 * another batch of characters is processed by the lexer until the next ambiguous situation occurs. This is
 * fundamentally different from the default behavior of Antlr.
 * </p>
 *
 * <h2>Automatic semicolon insertion (ASI)</h2>
 *
 * <p>
 * The EcmaScript specification mandates that valid implementations automatically insert a semicolon as a statement
 * delimiter if it is missing and the input file would become invalid due to the missing semicolon. That implies, that
 * not only valid implementations have to perform this, but a valid parser has to mimic this behavior in order to parse
 * executable code. The automatic semicolon insertion is implemented by two different means.
 * </p>
 *
 * <ol>
 * <li>The Antlr grammar that is generated by Xtext is post-processed to inject custom code into the grammar file before
 * it is passed to the Antlr tool.</li>
 * <li>The parser's error recovery strategy is customized so it attempts to insert a semicolon if it was expected.</li>
 * </ol>
 *
 * <p>
 * Both strategies have to work hand in hand in order to consume all sorts of legal JavaScript code.
 * </p>
 *
 *
 * <h3>Injected code in the Antlr grammar file</h3>
 *
 * <p>
 * Under certain circumstances, the parser has to actively promote a token to become a semicolon even though it may be a
 * syntactically a closing brace or line break. This has to happen before that token is consumed thus the rules for
 * {@link org.eclipse.n4js.parser.antlr.internal.InternalN4JSParser#ruleReturnStatement() return statements},
 * {@link org.eclipse.n4js.parser.antlr.internal.InternalN4JSParser#ruleContinueStatement() continue statements} and
 * {@link org.eclipse.n4js.parser.antlr.internal.InternalN4JSParser#ruleBreakStatement() break statements} are
 * enhanced to actively promote these tokens as semicolons.
 * </p>
 *
 * <p>
 * The same rule is applied to promote line breaks between an expression and a possible
 * {@link org.eclipse.n4js.n4JS.PostfixOperator}. At this location the line break is always treated as a semicolon
 * even though the operator may be validly consumed and produce a {@link org.eclipse.n4js.n4JS.PostfixExpression}.
 * </p>
 *
 * <p>
 * In both cases, the method {@code promoteEOL()} is used to move a token that may serve as an automatically injected
 * semicolon from the so called hidden token channel to the semantic channel. The hidden tokens are usually not handled
 * by the parser explicitly thus they are semantically invisible (therefore the term <em>hidden token</em>).
 * Nevertheless, they can be put on the semantic channel explicitly to make them recognizable. That's implemented in the
 * EOL promotion. The offending tokens include the hidden line terminators and multi-line comments that include line
 * breaks. Furthermore, closing braces (right curly brackets) are included in the set of offending tokens as well as
 * explicit semicolons.
 *
 * <h3>Customizing in {@link org.eclipse.n4js.parser.InternalSemicolonInjectingParser
 * InternalSemicolonInjectingParser}</h3>
 *
 * <p>
 * Since the EOL promotion does not work well with Antlr prediction mode, another customization complements that
 * feature. As soon as an invalid token sequence is attempted to be parsed and missing semicolon would make that
 * sequence valid, an offending token is sought and moved to the semantic channel. This is implemented in the custom
 * {@link org.eclipse.n4js.parser.InternalSemicolonInjectingParser#recover(org.antlr.runtime.IntStream, org.antlr.runtime.RecognitionException)
 * recovery strategy}.
 * </p>
 */
package org.eclipse.n4js.parser;
